Integrated analysis of content and construct validity

Establishing adequacy of psychometric properties of an instrument involves acquisition and evaluation of evidence based on item content and internal structure. Content validity evidence consists of subject matter experts providing quantitative ratings of the extent to which items are a representative sample of targeted domain. Evidence of internal structure includes factor analytic studies and examination of item interrelationships based on item responses from participants. Although subject matter expert ratings and participant response data are traditionally analyzed separately, each serves to inform the other in important ways. We propose integrating subject matter experts’ and participants’ data seamlessly to establish a unified model of validity evidence. The approach is applied to an instrument designed to measure nursing home culture change (i.e., resident-centered care). The proposed method has been demonstrated to be useful with a posterior distribution resulting in stable estimates of psychometric parameters superior to traditional analytic approaches. To illustrate the efficacy of the methodology, we present a simulation study and discuss its place in psychometric methods.